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Abstract. We give in this paper a convergence result concerning parallel 
asynchronous algorithm with bounded delays to solve a nonlinear fixed point 
problems. This result is applied to calculate the solution of a strongly mono- 
tone operator. Special cases of these operators are used to solve some problems 
related to convex analysis like minimization of functionals, calculus of saddle 
point and variational inequality problem. 
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1. Introduction 

In this paper we give a convergence result for parallel asynchronous iterations 
with bounded delays. The convergence result of these algorithms was shown by 
many authors. Chazan and Miranker in [6] treated the chaotic iterations using a 
linear and contractive mapping. In 1975, Miellou[9] extended the works of Chazan 
and Miranker to the nonlinear case using a contraction mapping and proposes a 
model with bounded delays. In 1978, Baudet in [3] generalizes the chaotic iterations 
of Chazan-Miranker and Miellou and proposes a model where the delays considered 
can be infinite. In a different context, El Tarazi[7] also established this result by 
a contraction technique according to a suitable scalar norm. Recently, Bahi[2] 
gave a convergence result concerning parallel asynchronous algorithm to solve a 
linear fixed point problems using nonexpansive linear mappings with respect to a 
weighted maximum norm. Our goal is to establish a convergence result concerning 
parallel asynchronous algorithm to solve a nonlinear fixed point problems using 
a nonlinear and nonexpansive mapping. All the results established in this study 
are mentioned by the author in [4]. We regard this study as a generalization to 
the asynchronous case of all results stated by Benahmed and Addou in [1] and 
so, we repeat the proofs given in [1] by including the modifications which requires 
the asynchronous case. Section 2 is devoted to a brief description of asynchronous 
parallel algorithm. In section 3 we prove the main result concerning the convergence 
of the general algorithm to a fixed point of a nonlinear operator from R" to R™. 
This result is applied in section 4 to the operator F = (I + cT) _1 (c > 0) which is 
called the proximal mapping associated with the maximal monotone operator cT 
(see Rockafellar[14]) to calculate a solution of the operator T. Special cases of these 
operators are also studied to solve optimization problems and variational inequality 
problem. 
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2. Preliminaries 

a 

R™ is considered as the product space Yl where a e N — {0} and n = 

i=l 

a 

n,i. All vectors x € R™ considered in this study are splitted in the form x = 

i=i 

(xi, ...,x a ) where Xi £ R ni . Let R ni be equipped with the inner product (., .) i and 
the associated norm ||..||j = (., .) V 2 . R™ will be equipped with the inner product 



(x,y) = ^2 ( x iiUi)i where x,y € R n and the associated norm ||x|| = (x,x) 



1/2 



i=l 



\\ x i\\i Y^ 2 - It will be equipped also with the uniform norm U^Uoo = max \\xi\\i. 

i=l l<i<a 

Definition 2.1. Define J = {J(p)} pgN a sequence of non empty sub sets of 
{l,...,a} and S = {(si(p), s a (p))} pgN a sequence o/N" and consider an op- 
erator F — (F\, ...,F a ) : R™ — > R". The asynchronous algorithm associated with F 
is defined by, 



(1) 



x? +1 



x° = {x° 1 ,...,x° a ) G R" 

x\ if iiJ(p) 

Fi{x{ l{p \...,x' a a{p) ) if ieJ( P ) 

i = 1, a 
I p = 0,l,.. 



ft will be denoted by (F, x° ,J,S). This algorithm describes the behavior of it- 
erative process executed asynchronously on a parallel computer with a processors. 
At each iteration p + 1, the i th processor computes x\ +1 by using (1). 
J(p) is the subset of the indexes of the components updated at the p th step. 
p— Si(p) is the delay due to the i th processor when it computes the i th block at the 
p th iteration. 

If we take Si(p) = p Vi € {l,...,a}, then (1) describes synchronous algorithm 
(without delay). During each iteration, every processor executes a number of com- 
putations that depend on the results of the computations of other processors in the 
previous iteration. Within an iteration, each processor does not interact with other 
processors, all interactions takes place at the end of iterations. 
If we take 

/ Si(p) =p VpeN,V«e {1, ...,<*} 

\ J{p) = {!,..., a} VpeN 

then (1) describes the algorithm of Jacobi. 
If we take 

s t (p)=p Vpe N,Vi e {1,. ..,a} 

J(p) = P + 1 (mod a) Vp € N 

then (1) describes the algorithm of Gauss-Seidel. 

For more details about asynchronous algorithms see [6], [9], [3], [7] and [5]. 

Definition 2.2. An operator F from R™ to R™ is said to be nonexpansive with 
respect to the norm ||..|| if, \\F(x) — F(x')\\ < \\x — x' \\ for all x, x' £ R" 



A CONVERGENCE RESULT FOR ASYNCHRONOUS ALGORITHMS AND APPLICATIONS 3 



3. The main result 

We establish in this section the convergence of the general parallel asynchronous 
algorithm with bounded delays to a fixed point of a nonlinear operator F : R™ — ► 
R™. 

Theorem 3.1. Suppose 

(ho) 3 a subsequence {pk}keN such that, Vi € {1, a}, i€ J(pk) and Si(pk) = Pk 
(hi) 3s g N, such that, Vi <G {1, ...,a} , Vp € N, p- s < Si(p) < p 
(h 2 ) 3u g R", F(u) = u 

(h 3 ) Va;, x' g R", ||F(ar) - F(a;')||oo < Hz-a^loo 

(/i 4 ) Vx,a;' g R", ||F(a;) - ^(x')!! 2 < (F(a;) - F(x'),x-x') 

Then, for all x° g R™ i/ie sequence (1) is convergent in R" to a fixed point x* of 
F. 

Proof. We follow the steps given in Addou-Benahmed [1] Theorem 4, with impor- 
tant modifications in the step (i). The steps (ii) and (Hi) are similar. We proceed 
then in three steps: 

(i) First, we show that the sequence {\\x p — «||oo} pe N ^ s convergent. For peN, 
we consider the (s + 1) iterates x p ,x p_1 , ...,x p ~ s in the process and put 

z p = max \\x p ~ l — ulloo = max \\x l — dl^ 

0<l<s p-s<l<p 

Then Vi g {1, a} we have, 
either i ^ J(p) so, 



^ 1 1 U 1 1 oo 

< max \\x p ~ l 

0<l<s 



or i g J(p) so, 

H^-UiHi - ||F i ( a: ; i W ) ... )a: '-W)-F i ( U )|| j 

< ||F(a;J l(p) ,...,^ (p) )- FHIU 

< ||K l(p) ,...,^ (p) )-"lloc (by (fc 3 )) 

= H^ 3 ^ — Uj\\j (for some j, 1 < j < a) 



< max \\x l — u||oo (use p — s < Sj (p) < p) 

p—s<l<p 



= Z p 



then 



Vie {!,...,<*}, \\x p+1 - u t \\ t < z p 
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that is 
therefore 



C P+1 -Ulloo < ZP 



Z p +1 = max \\xP +1 - 1 -ulloo 

0<i<s 

= Max < max ||x p ~' — ulloo , ||x p+1 — ulloo > 
\o</< s -i 11 " 11 11 J 

< Z p 

which proves that the sequence {z p } pgN is decreasing (positive) then it's con- 
vergent. It's limit is 

lim z p = lim max ||x p_ ' — u||oo 

p^oo p— »ooO<Ks 

= lim \\xP-3^ - ulloo (0 < j(p) < s) 
= lim ||x p - u||oo 

p^oo 

which proves that the sequence {||a; p — u||oo} peN is convergent and so, the 
sequence {£ p } peN is bounded. 
(ii) As the sequence {x Pk } kelS! is bounded ({pk} keN is defined by (ho)), it contains 
a subsequence noted also {x Pk } keN which is convergent in R" to an x*. We 
show that x* is a fixed point of F. For this, we consider the sequence {y p = 
x p — F(x p )} peN and prove that lim y Pk = 0. 

' k — >oo 

||x Pfc -u|| 2 = \\yP k + F(x Pk )-u\\ 2 

= \\y Pk \\ 2 + \\F(x Pk ) - "II 2 + 2(F(x Pk ) - u,yP k ) 

then 

||yP*||2 = \\ xPk _ u ||2 _ \\ F ( x p*) - u f -2{F(x Pk ) -u,y Pk ) 
however 

(F(xP k ) -u,yP k ) = (F(xP k ) - F(u),xP k - F( X P k )) 

= (F(x Pk ) - F(u), [xP k - F(u)\ - [F(x Pk ) - F(u)}) 
= (F(x Pk ) - F(u),x Pk -u) - \\F( X P k ) - F(u)\\ 2 
> (by (hi)) 

so, 

||y»||2 < \\ xPk - u \\ 2 -\\F(xP k )-u\\ 2 

= \\xP k - u\\ 2 - WXPX+ 1 - u\\ 2 (by (ho)) 
However, by (i) the sequence {||a; p — w||oo} pgN is convergent, then the se- 
quence {||a; p — u||} p£N is also convergent with limit 

lim ||a; p — u|| = lim ||.T Pfc — u| 

p^oo k^oo 

= lim ||.T Pfc+1 — u| 

k^oo 

= \\x* - u| 



and so 

which implies that 



lim ||y Pfc || = 

k^OO 



lim y Pk = 

k^oo 
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and so 

x* -F(x*) = 
that is x* is a fixed point of F. 
(Hi) We prove as in (i) that the sequence {\\x p — ^*||oo} pe N ^ s convergent, so 

lim \\x p — x*\\ oc = lim \\x Pk — x*||oo = 

Which proves that x p — > x* with respect to the uniform norm ||..||oo- ■ 

Remark 3.2. The hypothesis (ho) means that the processors are synchronized and 
all the components are infinitely updated at the same iteration. This subsequence 
can be chosen by the programmer (Bahi[2]). 

Remark 3.3. The hypothesis (hi) means that the delays dues to the communica- 
tions between processors and to the calculus are bounded, which means that after 
(s + 1) iterations, all the processors are supposed to have update their own data 
(Bahi[2\). 

Remark 3.4. The hypothesis (h^) is verified by a large class of operators. For 
example, the resolvent F\ = (I + AT) -1 (where X > 0) associated with a maximal 
monotone operator T (see Lemma 4-3 below). Again, the metric projection p c of a 
Hilbert space H onto a nonempty closed convex set C; that is, for x G H , p c (x) is 
the unique element of C which satisfies 

\\x -p c (x)\\ = inf || a? — 
yec 

see for proof, Phelps[10], Examples 1.2. (f). In the linear case, take for example a 
linear operator which is symmetric, positive semi-definite (or simply positive) and 
nonexpansive, as shown in the following proposition: 

Proposition 3.5. Let A be a linear symmetric positive and nonexpansive operator 
in R™. Then A verify the hypothesis (hi). 

Proof. Recall that an operator A is said to be symmetric if for all x,y £ R™, (Ax, y) = 
(x,Ay). 

(i) The operator B = I — A is symmetric. Indeed, Vx, y G R™ 

(Bx,y) = (x- Ax,y) = (x,y) - (Ax,y) 
= (x,y) - (x,Ay) = (x,y- Ay) 
= (x,By) 

(ii) The operator B est positive. Indeed, Vie G R" 

(Bx, x) = (x - Ax, x) = \\x\\ 2 - (Ax, x) > 

since (Ax,x) < ||Ar||||x|| < ||a;|| 2 . 
(Hi) A and B are commuting operators. Indeed, 

AB = A(I - A) = A- A 2 = (I - A) A = BA 

(iv) AB is a symmetric operator. Indeed, \/x,y G R" 

(ABx, y) = (Bx, Ay) = (x, BAy) = (x, ABy) 

(v) AB is a positive operator (see proof in [8], Theorem 10.7). 
(vi) The operator A verify the hypothesis (h^). Indeed, Vx G R" 

(Ax, x) - \\Ax\\ 2 = (Ax, x - Ax) = (Ax, Bx) = (ABx, x) > 
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4. Applications 

4.1. Solutions of maximal strongly monotone operators. In this section, we 
apply the parallel asynchronous algorithm with bounded delays to the proximal 
mapping F = (I + cT) _1 (c > 0) associated with the maximal monotone operator 
cT. We say that a multifunction T from D(T) C R™ — » R™ is monotone if 

Vx, x' e D(T), (x -x',y- y') > 0, Vy G Tor, Vj/' € Tx'. 

It is said to be maximal monotone if, in addition, the graph 

G(T) = {{x, y) : x G D{T) and y G Tx} 

is not properly contained in the graph of any other monotone operator T" : D(T) C 
R" — > R". It is said to be strongly monotone with modulus a (a > 0) or a-strongly 
monotone if 

Vx,x' e £>(T), (x-x',y-y') > a||a;-x'|| 2 , Vy e Tx, Vy' e Tx'. 

Let T be a multivalued maximal monotone operator defined from R™ to R™. A 
fundamental problem is to determine an x* in R" satisfying G Tx* which will be 
called a solution of the operator T. The following Theorem gives a general result 
concerning the solution of a maximal strongly monotone operator. 

Theorem 4.1. Let T be a multivalued maximal a-strongly monotone operator in 
R" (a > 0).Then 

(1) T has a unique solution x* . 

(2) Any parallel asynchronous algorithm with bounded delays associated with 
the single-valued mapping F = (I + cT) -1 where c > v/ °~ 1 converges in 
R™ to the solution x* of the problem G Tx. 

Proof. We give the proof in the form of Lemmas. The two Lemmas 4.2 and 4.3 
were shown in [1] by Addou and Benahmed. 

Lemma 4.2. (Addou- Benahmed [1], Theorem 4) Let T be a maximal monotone 
operator in R™ and F = (I + cT) -1 , (c > 0). Then the solutions of T are exactly 
the fixed points of F in R" . 

Proof of Lemma 4.2. 

G Tx x G {I + cT)x 

x={I + cT)~ 1 x 
x — Fx 

■ 

Lemma 4.3. (Addou- Benahmed [1], Theorem 4) Let T be a maximal monotone 
operator and F = (I + cT)^ 1 (c > 0). Then F satisfy the hypothesis (hi). 

Proof of Lemma 4. 3. Consider x,x' in R™ and show that 

|| J F(x)- J F(x')|| 2 < (F(x)-F{ 

Put y = F{x) and y 1 = F(x') then, 

( x G y + cTy 

\ x' G y' + cTy' 

i.e. 

f x - y G cTy 

\ x'-y' £ cTy' 
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As c > 0, the operator cT is monotone and so, 

((x-y)-(x , -y'),y-y') >0 

therefore 

(x-x',y-y') - \\y-y'\\ 2 > 

which implies 

||F(a;) - F{x')\\ 2 < (F(x) - F(x'), x - x') 

■ 

We complete the proof of Theorem by the Lemma, 

Lemma 4.4. Let T be a maximal a-strongly monotone operator in R™ (a > 0) and 

F = {I + cT)- 1 (c > 0). Then 

(a) F has a unique fixed point x* . 

(b) For c > v ^~ 1 , the mapping F is nonexpansive with respect to the norm \\..\\oo 
inW 1 . 

Proof of Lemma 4.4. 

(a) Consider T = T-aI, = ^ and F' = (I + f3cT')- 1 . Take x G R", 

y = F(x) x e y + cTy 

■<=> x G (1 + ac)y + c(T - al)y 
(3x G (3{1 + ac)y + ficT'y 
(3xe(I + /3cT')y 
y = F'(px) 

i.e.,VxGR™, F(x) =F'(f3x). 

As T is a-strongly monotone (maximal) , the operator T' is maximal monotone 
and then the map F' — (I + ftcT')^ 1 is nonexpansive in R™, so Vx, x' G R" 

\\F(x) - F(x')\\ = \\F'(/3x) - F'(px')\\ < f3\\x - x'\\ 

^ s $ = i+ac < ^' * nc application F is contractive in R™ and then has a 
unique fixed point x* (Banach's fixed point Theorem) which will be the solution 
of the operator T by Lemma 4.2. 

(b) The Euclidean norm and the uniform norm are equivalents in R" by the rela- 
tion: 

IMloo < IMI < V^IMloo , Vx G R" 

Then, Vx, x' G R" 

\\F(x)-F(x')\\<0\\x-x'\\ 

implies 

\\F(x) - Fix')]^ < /V^lz-z'IU 

Va I 
1+ac I 

It is sufficient to take c such that 



1 + ac 
i.e. 

(2) c>^± 

a 

So, the theorem is entirely shown. 
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4.2. Minimization of functional. Let's begin by this proposition that provides 
a characterization of strongly convex functions. Recall that a function / : R™ — > 
R U {+00} is said to be strongly convexe with modulus a (a > 0) or a-strongly 
convex if for all x, x' G R™ and t g]0, 1[ one has 

f(tx + (1 - t)x') < tf(x) + (1 - t)f{x') - ±at(l - t)\\x - x'\\ 2 

The subdifferential of a proper (i.e not identically +00) convex function / on R" 
is the (generally multivalued) mapping df : R" — ► R" defined by 

df(x) = {ye R" f{x') > f(x) + (y, x' - x) , W G R"} 

which is a maximal monotone operator if in addition / is a lower semicontinuous 
function (lsc). 

Proposition 4.5. (Rockafellar[14:, Proposition Q\) Let f : R™ RU {+00} be 
convex proper and lower semicontinuous. Then the following conditions are equiv- 
alent: 

(a) f is a-strongly convex 

(b) df is a-strongly monotone 

(c) whenever y G df{x) one has for all x' G R": 

>/(*) + -*> + ^11*' - *ll 2 

Corollary 4.6. Let f : R™ RU {+00} be a lower semicontinuous a-strongly 
convex function which is proper. Then 

(1) f has a unique minimizer x* . 

(2) Any asynchronous parallel algorithm with bounded delays associated with 
the single-valued mapping F = (7 + cdf)^ 1 where c > ~^ converges to 
the minimizer x* of f in R" . 

Proof. Remark that 

G df(x) <^=> f{x') > f(x) W G R" 
<^=> f(x) = min f[x') 

so, the solutions of the operator df are exactly the minimizer of /. 

The subdifferential df is maximal and a-strongly monotone (Proposition 4.5). We 

apply then Theorem 4.1 to the operator df. ■ 

4.3. Saddle point. In this paragraph, we apply Theorem 4.1 to calculate a saddle 
point of functional L : R" x R m — » [—00, +00]. Recall that a saddle point of L is 
an element (x*,y*) of R" x R m satisfying 

L(x*,y) < L(x*,y*) < L{x,y*), \/(x,y) G R" x R m 

which is equivalent to 

L(x*,y*)= inf L(x,y*) = sup L(x* ,y) 
xeR" yen™ 

Suppose that L(x, y) is proper and convex lower semicontinuous in x G R n , concave 
upper semicontinuous in y G R m , then L is a proper closed saddle function in the 
terminology of Rockafellar[ll]. Let the subdifferential of L at (x, y) G R™ x R m , 
dL(x,y), be defined as the set of vectors (z,i) G R" x R m satisfying 

V(x', y') G R" x R m L(x, y') - (?/ - y, t) < L(x, y) < L(x', y) - (x' - x, z) 
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then the multifunction Tl defined in R™ x R m by 

T L (x, y) = {(z, t) G R" x R m : (z, -t) G dL(x, y)} 

is a maximal monotone operator; see Rockafellar[ll],[12]. In this case the global 
saddle points of L (with respect to minimizing in x and maximizing in y) are the 
elements (x,y) solutions of the problem (0,0) G Ti,{x,y). That is 

(0, 0) € Tl(x* , y*) (x*,y*)=arg min maxL(x,y) 

Definition 4.7. The functional L from R" x R m to [— oo, +oo] is said to be strongly 
convex-concave with modulus a (a > 0) or a-strongly convex- concave if L{x,y) is 
a-strongly convex in x and a-strongly concave in y. 

Lemma 4.8. // the functional L is a-strongly convex-concave, then the multifunc- 
tion Tl is an a-strongly monotone operator. 

Proof. Define the inner product and the norm in R" x R m as follows: For 
(x,y),(x',y')€B. n xTL m : 

f ((x, y), (x',y')) Rnx Rm = (x, x') Rn + (y, y') nm 
\ \\(x,y)\\Rn xR m = v/|jx|j^„ + ||y||^ m 

we write simply as 

f ((x,y),(x', y ' )) = (x,x') + (y,y') 
\ \\(x,y)\\ = v / INI 2 + IMI 2 

Consider (x,y), (x' ,y') in R™ x R m , (z,t) e T L (x,y) and (z',t') G T L (x',y') and 
show that 

((x, y) - (x',y f ), (z, t) - (z', t')) > a\\ (x, y) - (x' , y')l| 2 
The function L(x,y) is a-strongly convex in x. Proposition 4.5 implies that the 
operator d x L is a-strongly monotone in x. As z G d x L(x, y) and z' G d x L(x', y') 
we obtain, 

(z — z', x — x 1 ) > a\\x — x'\\ 2 

In the same way, —t G d v L(x, y), —t 1 G d y L(x', y') and d y (—L) is a-strongly mono- 
tone in y (use proposition 4.5 with f(y) = —L(x,y)) we obtain, 

((-t)-(-t'),y-y') <-a\\y-y'f 

thus is 

{t-t',y-y')>a\\y- y r 

therefore 

(z -z',x- x 1 ) + (t-tf,y- y') > a(\\x - x'\\ 2 + \\y- y'|| 2 ) 

((z,t) (z',tf),(x,y) - (x',y')) > a\\(x,y) (x',y')l| 2 
which proves that Tl is a-strongly monotone in R™ x R m . ■ 

If L is a a-strongly convex-concave function and proper closed then Tl is maxi- 
mal (see Rockafellar[12]) a-strongly monotone (Lemma 4.8). We can then apply 
Theorem 4.1 to the operator Tl so, 

Corollary 4.9. Let L be a proper closed a-strongly convex- concave function from 
R" x R m into [-oo,+oo]. Then 
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(1) L has a unique saddle point (x*,y*). 

(2) Any parallel asynchronous algorithm with bounded delays associated with 
the single-valued mapping F = (I + cTl)^ 1 where c > — 1 from R n x R m 
into R" x R" 1 converges to the saddle point (x*,y*) of L. 

4.4. Variational inequality. Let C be a nonempty closed convex set in R" and 
A a multivalued maximal monotone operator in R™ such that D(A) = C. The vari- 
ational inequality problem in its general form consists of finding x* G C satisfying 

(3) 3 y* G Ax*, (y*,x-x*) > 0, Va; G C. 

For x G R n , let N c (x) be the normal cone to C at x defined by 

N c (x) = {t/e R" :(y,x-z)> 0, VzeC}. 

The multifunction T defined in R™ by, 

f Ax + N c (x) tfxeC 
w \ <t> ifx(£C 

is a maximal monotone operator (Rockafellar[13]). 

Lemma 4.10. If A is a-strongly monotone then T is an a-strongly monotone op- 
erator. 

Proof. Consider x, x' G D(T) = C,y e Tx and y' G Tx 1 
then 

y = yi + U2, yi e Ac, 2/2 e N c (a;) 
y' = yi + 2/ 2 > e Ar', ^ g A^a/) 

however 

j/2 G iV c (a;) (y 2 ,z - 0} > 0, VzeC 
^GA^aO^O/^'-z^O, VzeC 

therefore 

(y-y',x-x') = (y 1 -y' 1 ,x-x') + (y2-y' 2 ,x-x') 

= (yi-y[,x- x') + (y 2 , x-x') + (y' 2 ,x' - x) 

S v ' " v ' V v ' 

>a||x-x'|| 2 >0 >0 

- a||a;-x'|| 2 



Lemma 4.11. The solutions of the operator T are exactly the solutions of the 
variational inequality problem (3). 

Proof. 

e Tx* <=4> e Ax* + N c (x*) 



By* e Ax* 
By* G Ax* 
3y* G Ax* 
By* G Ac* 



0€y* +JV c (a;*) 
-y* G JV c (a;*) 
(-y*,a:* - z) > Vz e C 
(y*,z-x*) > Vz G C* 



a;* is solution of (3) 



By applying the Lemma 4.10, 



Lemma 4.11 and Theorem 4.1, we can write 
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Corollary 4.12. Let C be a nonempty closed convex set in R" and A a multivalued 
maximal a-strongly monotone operator in R" such that D{A) = C. Then 

(1) The variational inequality problem (3) has a unique solution x* . 

(2) Any parallel asynchronous algorithm with bounded delays associated with 
the single-valued mapping F = (I + cT)~ x where T defined by (4) and 
c > converges to the solution x* of the problem (3). 
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